Systems, methods, and devices for highly interactive large image display and manipulation on tiled displays

ABSTRACT

Display units which can be arranged to form a single arrayed display system, thereby allowing the display of much larger images than can be shown on a single display. Each display unit can include an image display, a communication mechanism, such as a network interface card or wireless interface card, and an image display module, such as a video card and an image processor. Each display module can be configured to selectively process and display portions of large digital images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 12/545,026, filed Aug. 20, 2009, which claimspriority to U.S. Provisional Patent Application No. 61/090,581 filed onAug. 20, 2008, the entire contents of which is expressly incorporated byreference herein.

BACKGROUND

1. Field

This disclosure generally relates to visualization technologies. Morespecifically, this disclosure relates to display devices which can beused to form a display system formed of tiled arrays of display devices.

2. Description of the Related Art

Traditionally, personal computers and workstations are connected to oneor a small number of adjacent display devices, often LCD type monitors.Such systems can provide the user with the ability to view a largernumber of pixels than that typically displayable on a single monitor.

Commercially available computer systems can often support one or twomonitors for each video controller (sometimes constructed in the form ofa “video card”) connected to the system. For example, typical “PC”computer systems include several “expansion slots” which can acceptcertain types of video cards. Motherboards of some “PCs” are built withone or more PCI, PCI Express, AGP, etc., slots that can accept videocards. In this manner, a single computer can be provided with multiplevideo cards to increase the number of displays that can be controlled bythe computer.

SUMMARY

The present disclosure relates to methods and systems for displaying andmanipulating large images or datasets. Embodiments of the presentdisclosure can be particularly advantageous for displaying large digitalimages that can be tens or hundreds of millions of pixels, or evenbillions of pixels, on tiled array display systems in a highlyinteractive manner. Some embodiments can allow for display of datasetsthat can be over a gigabyte in size. In some embodiments, a system isdisclosed that can display a large image on a tiled array display thatincludes one or more display units and then can manipulate the image bypanning, zooming, rotating, color filtering, and the like.

Some embodiments of the systems disclosed herein can be configured insuch a way that each of the display units of the system retain a fullcopy of an original image file (e.g., in local memory) to be displayed,then in parallel, each display unit processes the original image fileand displays only that portion of the original image corresponding tothe position of the display unit in the overall array. For example, insome embodiments, the display units can be provided with a configurationparameter, indicating in which part of the array the corresponding unitis positioned. For example, in an array of 20 display units formed of 4rows and 5 columns, the display unit in the upper left hand corner canbe identified as the column 1, row 1 display device. The positions ofthe display units can also be correlated to the ranges of rows andcolumns of pixels each display unit can display, for example, at theirrespective “native” resolutions, together forming a “single display”including all of the pixels from each display device.

The display units can also be configured to receive a position commandassociated with an image file located in each units' local memory. Theposition command can be any data indicative of the position of theassociated image file on the array. For example, the position commandcan be an identification of the desired location of a predeterminedpixel in the associated image. Such a predetermined pixel, for example,can be the pixel at the center of the image that would result from theassociated image file being processed at its native resolution. Otherpredetermined pixels can also be used. Additionally, the positioncommand can include other display characteristics, such as the desiredrotational angle of the displayed image, magnification, resolution, etc.

In some embodiments, the display units can be configured to use thereceived position command to determine what portion (if any) of anassociated image corresponds to the pixels of that display unit. Forexample, the display unit can be configured to determine the requestedlocation of the predetermined pixel of the image resulting fromprocessing of the associated image file, for example, with reference toa virtual wireframe representation of the overall array. Then thedisplay unit can resolve the image file to then determine what portionof the image (if any) corresponds to the display on that display unit.

After obtaining the result of this initial analysis, the display unitcan process the image file and display the discrete portion of theresulting image in the correct orientation and thereby, together withthe other image portions displayed by the other units in the array,display an observable mosaic of the original image.

As noted above, by configuring the display units to receive a basicposition command and then, in parallel, process the position command,process the image file, and display the corresponding portion of theimage, the system can generate the desired view of the image morequickly than a system in which an image is broken down and pre-processedinto individual parts, which are then individually distributed to thecorresponding display unit. Instead, in some embodiments disclosedherein, as the display units receive a series of position commands,which request the display of an image stored on the local memory of eachdisplay unit at different locations on the array, all of the displayunits can re-process and display their respective image portions muchmore quickly, thereby providing the user with a more responsive anduseful system.

Some embodiments of the systems disclosed herein can also allow thedisplay of one or more images that can be larger than the memory of theindividual display units in the tiled array display can handle.Conventionally, multiple images can be prohibitively expensive to loaddue to, for example, disk access times of loading hundreds of megabytesor gigabytes, and a lack of available memory. Advantageously, in someembodiments, each display unit in the tiled array display can beconfigured to only load or process the data needed for its local portionof the overall image displayed on the tiled array display. The systemcan thus greatly reduce the amount of data to load into memory, forexample, and allows rapid manipulation of the image portions on theindividual display units. If more data is needed, the required data canalso be loaded and displayed. The system can advantageously employsparallel processing techniques, such as multithreading, to improvesystem performance.

In some embodiments, each full size or original image can bepreprocessed and stored in a hierarchical format that includes one ormore sub-images, where each sub-image can be a reduced size or reducedresolution version of the original image. The largest sub-image can bethe same size as the original image and/or include image content fromthe original image. In some embodiments, each sub-image can be stored asone or more blocks, or tiles, to allow rapid access to a particular partof the image without having to access entire rows or columns of pixels.This can advantageously allow a display unit of the tiled array displayto determine the discrete portion of an image it needs (e.g., has beenrequested) to display and to fetch exactly the level of detail, forexample the sub-image, and/or to quickly fetch the needed blocks, ortiles, that make up the portion of the image displayed on the displayunit. In other embodiments, each sub-image can be stored as one or moreresolution layers to allow for rapid access to, and display of, aparticular resolution of the original image.

This can be particularly beneficial because only the portion of eachoriginal image required for each tile of the tiled array display needsto be resident in the memory of the respective display node. Thus, onlythe blocks or resolution layers of the appropriate sub-image may need tobe loaded, which improves responsiveness of the tiled array displaysystem and performance of the individual display units. In someembodiments, surrounding blocks and blocks from higher and lower levelsin the hierarchy can also be pre-fetched for improved performance. Thiscan be advantageous for enabling each tile node to support the displayof more than one original image or portions thereof. Additionally, aresource management approach can be used to support increasedinteractivity by reducing the amount of data loaded into memory, forexample, and/or allowing portions of several images to be resident oneach individual tile display unit, which supports the display andmanipulation of multiple full size images on the array display.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned and other features of the inventions disclosedherein are described below with reference to the drawings of preferredembodiments. The illustrated embodiments are intended to illustrate, butnot to limit the inventions. The drawings contain the following Figures:

FIG. 1A is a schematic representation illustrating an embodiment of ahighly interactive tiled array display.

FIG. 1B is a schematic diagram illustrating an embodiment of aninteractive tiled display system for displaying and manipulating one ormore images on an array-type display.

FIG. 2 is a flow chart illustrating an embodiment of a method forprocessing one or more images for display on an array display.

FIG. 3 is a flow chart illustrating an embodiment of a method fordisplaying an image on an array display.

FIG. 4 is a flow chart illustrating an embodiment of a method forcontrolling the display of one or more images on an array display.

FIG. 5A schematically illustrates an image overlapping two display nodesin an array display.

FIG. 5B schematically illustrates the image of FIG. 5A partitioned overtwo display nodes.

FIG. 6 is a schematic diagram of a tiled array display and a controlnode, the control node including a user interface having a schematicrepresentation of the tiled array display.

DETAILED DESCRIPTION

The present disclosure generally relates to the interactive display andmanipulation of large images or large datasets on an array-type display,such as a tiled display system. In some embodiments, a system thatimplements a highly interactive large image or parallel display systemcan be used. In some of the embodiments described below, the interactivetiled display system comprises a high resolution, large format displaysystem configured to render large-scale images or datasets on a largearray of display units, or “nodes.” The interactive tiled display systemadvantageously allows a viewer to see minute, high-resolution detail inthe context of a large overall image.

In some embodiments, the interactive tiled display system allows a userto interact with the display in real-time. For example, embodimentsdescribed below can allow panning, zooming, resizing, cropping,rotating, color shading, transparency controlling, and the like ofimages and/or other content on the tiled display, thereby enabling usersto examine content with increased flexibility and effectiveness.

The interactive tiled display system can employ parallel processingtechniques (e.g., multithreading) to increase the speed with which thelarge image or dataset is displayed and/or manipulated on the tiledarray display. The use of parallel processing techniques canadvantageously result in increased scalability without sacrificingperformance.

In some embodiments, the interactive tiled display system comprises asymmetric multiprocessing (SMP) system, wherein each of the tileddisplay units share the same memory, system bus, and input/output (I/O)system. In other embodiments, the interactive tiled display systemcomprises a massively parallel processing (MPP) system, wherein each ofthe display units, or nodes, has its own memory, bus, and I/O system.

In yet other embodiments, the interactive tiled display system comprisesa clustered system, wherein each of the display nodes are coupled usinglocal area network (LAN) technology and each of the display nodescomprises an SMP machine. In still other embodiments, the interactivetiled display system comprises a distributed memory parallel processingsystem, such as a non uniform memory access (NUMA) or distributed sharedmemory system. In some embodiments, the interactive tiled display systemcomprises a global shared memory system, such as a uniform memory access(UMA) system. In other embodiments, the interactive tiled display systemcomprises a cache only memory architecture (COMA) system. In still otherembodiments, the interactive tiled display system comprises a parallelrandom access machine (PRAM) system. Other types of parallel processingsystems or modified versions of the above systems can also be usedwithout departing from the spirit and/or scope of the disclosure.

In some embodiments, the interactive tiled display system can comprise,without limitation, one or more of the following parallel processingarchitectures: linear array, ring array, binary tree, 2D mesh, torus,shared-memory, and hypercube.

In some embodiments, the nodes forming the interactive tiled displaysystems described below can identify and process a discrete portion of afull image to be displayed on the array. This can reduce the amount ofprocessing required by a control node used to control the nodes, andthus can increase the responsiveness of the overall tiled displaysystem. The interactive tiled display system can also allow for movementof images around the tiled display at a rate that is much faster thanother techniques. For example, the interactive tiled display system canbe configured to dynamically set up binary trees to increase the speedwith which information or data is communicated to the display nodes. Thenumber of binary trees can be dynamically modified (e.g., increased ordecreased) depending on communication received from the control node.

Some of the embodiments described below can accommodate viewing ofmultiple highly detailed, large images, which can exceed billions ofpixels, to be displayed as part of a high resolution, coordinatedworkspace on a tiled display. The real-time or near real-timeinteraction with the multiple image data, which can be received frommultiple image data sources, can include panning, zooming, rotating,color filtering, and transparency control of the images more quickly.The interactive tiled display system can be beneficial for viewing orvisualizing various types of image data, such as medical, cancer cells,satellite, geosciences, oil monitoring, weather monitoring orprediction, traffic control, astronomy, artwork, and the like.

In some embodiments, a control unit can function as a front end userinterface to allow a user to control the placement and manipulation ofcontent on the tiled array display via user interaction devices (e.g.,keyboard, mouse) associated with the control unit. For example, thecontrol unit can include a graphical user interface having a displaythat “mirrors” the tiled array display. Manipulation of an image on thegraphical user interface on the control unit can be used to controlmanipulation of the image on the tiled array display. Such systems andmethods can be useful when a user desires to display an image that islarger than a traditional display connected to a user computer canhandle.

Embodiments are described below with reference to the accompanyingfigures, wherein like numerals refer to like elements throughout. Theterminology used in the description presented herein is not intended tobe interpreted in any limited or restrictive manner, simply because itis being utilized in conjunction with a detailed description of somespecific embodiments of the invention. Furthermore, embodiments of theinventions may include several novel features, no single one of which issolely responsible for its desirable attributes or which is essential topracticing the embodiments herein described.

FIG. 1A is a schematic representation illustrating an embodiment of ahighly interactive tiled array display 10. The interactive tiled arraydisplay 10 can comprise a plurality of display units 100, wherein eachdisplay unit 100 is configured to determine a respective portion of thelarge image to be output on its associated display and to process anddisplay the respective portion in parallel. For example, the interactivetiled array display 10 includes a 5-by-5 array of display units. In someembodiments, the highly interactive tiled array display 10 comprises a10-by-5 array of display units (each comprising a 2560×1600 pixeldisplay) to form a 25,600×8,000 pixel display. However, the number ofdisplay units 100, and thus, the size of the tiled array is not limitedto any particular array size, and can be expanded as large as space andnetwork bandwidth permit.

FIG. 1B is a block diagram illustrating an interactive tiled displaysystem 50 for displaying and manipulating one or more images on anarray-type display, such as the highly interactive tiled array display10, which can be “parallelized.” The interactive tiled display system 50can comprise a plurality of display nodes, or display units 100(including display nodes 100A, 100B, and 100N that are representative ofany quantity of display nodes) that are in communication with a network160 and other devices via the network 160, including a control node 102.In alternative embodiments, the plurality of display units 100 can beconnected in a serial configuration (e.g., a daisy chain implementation)instead of a parallel implementation.

In the illustrated embodiment, an original image data source 164, whichcan be, for example, a storage device (e.g., a network-attached storagedevice, a RAID storage system, or a shared local storage over parallelfile system, such as a parallel virtual file system (PVFS)) or acomputing device, is also in communication with the network 160. In someembodiments, the original image data source 164 comprises a sharedmemory storage device/file system that is accessible by each of thedisplay nodes 100 via the network 160. The image data source 164 cancomprise a plurality of preloaded content, such as large,high-resolution digital images and/or datasets.

Generally, the control node 102 can comprise one or more computerdevices that gather or make available information about the state of theoverall tiled display system 50, including the display nodes 100,through the use of messages. As used herein, “messages” can be any typeof message including data, packets, signals, state values, displayparameters, etc., and are referred to interchangeably below as“messages” or “state messages.” In some embodiments, the messagescomprise “global” messages, which can be broadcast to all the displaynodes 100 by the control node 102 via the network 160. The control node102 can also be configured to transmit such messages to a subset of oneor more of the display nodes (e.g., via multicast or unicast messagetechniques).

The messages can include information about a current state of an imageto be, or being, displayed on the tiled array display. For example themessages can include information about a location, size, resolution,orientation, color scheme, or identification of the image. In someembodiments, the messages comprise image data or other data content.

In some embodiments, the communication of messages to the display nodes100 can be controlled by a multi-port Gigabit Ethernet switch; howeverother switches, hubs, or routers can also be used. The control node 102can comprise a desktop computer, laptop, tablet, notebook, handheldcomputing device (e.g., a smartphone or PDA), a server, or the like. Insome embodiments, the control node 102 comprises one or more multi-coreand/or multiprocessor computing systems configured for use inimplementing parallel processing techniques, such as MIMD or SPMDtechniques using shared and/or distributed memory. In addition, thecontrol node 102 can function as a front end user interface to theinteractive tiled display system 50 that allows a user to interact withthe overall system by manipulating the image or images displayed on itsassociated display, which in turn manipulates the image or images on thetiled display. Such functions are described in more detail below.

Any of the display nodes 100N and/or the control node 102 can be used toimplement the systems and methods described herein. For example, in someembodiments, the display node 100A and the control node 102 can beconfigured to manage the display of information on tiled displaysystems. In some embodiments, the control node 102 and the display nodes100 are configured to implement a shared memory MIMD system and/or amessage passing MIMD system. The functionality provided for in thecomponents and modules of the display node 100A and the control node 102can be combined into fewer components and modules or further separatedinto additional components and modules.

With continued reference to FIG. 1B, although only exemplary componentsof the display node 100A are described in detail, it is to be understoodthat the descriptions of the display node 100A set forth herein alsoapply to the other nodes 100B, 100N.

In some embodiments, each of the display nodes 100 is configured torender and display a portion of a large, high-resolution image. In someembodiments, all the display nodes 100 work in parallel to render thetotal overall image across the plurality of display nodes 100, therebyavoiding the performance limitations that would arise from dividing andrendering the entire high resolution image into discrete parts on asingle computing device, then transmitting the discrete parts to thecorresponding nodes of the tiled array. In some embodiments, the displaynodes 100 can display digital images or other data content of 1 gigabyteor larger in size.

In some embodiments, the display node 100A can include, for example, acomputing device, such as a personal computer (PC), that is IBM,Macintosh, or Linux/Unix compatible. In some embodiments, the computingdevice comprises a server, a laptop computer, a monitor with a built-inPC, a cell phone, a personal digital assistant, a kiosk, or an audioplayer, for example.

In some embodiments, the display node 100A includes a central processingunit (“CPU”) 105, which can include one or more multi-core processors,microprocessors, graphics processors, digital signal processors, and/orthe like. The display node 100A can further include a memory 130, suchas random access memory (“RAM”) for temporary storage of information anda read only memory (“ROM”) for permanent storage of information, and amass storage device 120, such as one or more hard drives, diskettes,and/or optical media storage devices. Other arrangements of memorydevices can also be used. Thus, the display node 100A can be consideredas having a “memory system.” Thus, as used herein, the term “memorysystem” can comprise only one or any combination of the memory device130, mass storage device 120, processed image data source 162, and anynumber of additional memory devices 130, mass storage devices 120,processed image data sources 162, or any other type of memory.

Typically, the modules of the display node 100A are connected to the CPU105 using a standards-based bus system. In different embodiments, thestandards-based bus system can be Peripheral Component Interconnect(PCI), Microchannel, SCSI, Industrial Standard Architecture (ISA) andExtended ISA (EISA) architectures, for example. Other types of systemscan also be used. Using any of the above or other systems provides foran operable connection between the devices 105, 110, 120, 130, 140, 150,155, 162, 166, any memory system as described above, or other devices.

The box illustrated in FIG. 1B and identified with the reference numeral100A, can be considered as a schematic representation of a housing ofthe display node 100A. Such a housing can be formed in any manner, suchas those designs currently used on commercially available monitors ortelevisions, including LCD, plasma, LED, or other types of devices. Sucha housing can be shaped to enclose the devices noted above, for example,enlarged if necessary. Such shaping is fully within the skill of one ofordinary skill in the art of video monitor or television design.

The housing can also include a mount, such as the mounting hardwarenormally included on the rear sides of the LCD, plasma, LED monitors andtelevisions that are currently widely available on the commercialmarket. Such mounts can be described as “wall mounts.” Other hardwarecan also be used. Such hardware can be used in conjunction with anappropriately shaped rack designed to connect to and support a pluralityof display nodes 100A having such mounts on their respective housingswith the plurality of display nodes 100A arranged adjacent each other,in a tiled layout, such as the layout schematically represented in FIG.1A.

The display node 100A is generally controlled and coordinated byoperating system software, such as Windows 95, Windows 98, Windows NT,Windows 2000, Windows XP, Windows Vista, Linux, SunOS, Solaris, areal-time operating system (RTOS), MAC OS X, or other compatibleoperating systems. In other embodiments, the display node 100A may becontrolled by a proprietary operating system. The operating systemscontrol and schedule computer processes for execution, perform memorymanagement, provide file system, networking, and I/O services, andprovide a user interface, such as a graphical user interface (GUI),among other things.

The display node 100A can include one or more input/output (I/O) devicesand interfaces 110, such as a keyboard, a mouse, a touchpad, a scrollwheel, a trackball, a voice activation unit, a haptic input device, aprinter, or I/O ports. In addition, the display node 100A can includeone or more display devices 166, such as a monitor, that allows thevisual presentation of data, such as the image data described herein, toa user. In some embodiments, the display device 166 can comprise an LCDdisplay, such as a 30-inch LCD Cinema Display available from Apple, Inc.or a 46-inch 460UXn-UD LCD display available from Samsung Electronics.In other embodiments, the display device 166 comprises a plasma, CRT, orOrganic LED display. In yet other embodiments, the display device 166comprises a projector-based display. In some embodiments, the displaydevice 166 provides for the presentation of scientific data, GUIs,application software data, and multimedia presentations, for example.The display node 100A may also include one or more multimedia devices140, such as speakers, video cards, graphics accelerators, cameras,webcams, and microphones, for example.

In some embodiments, the I/O devices and interfaces 110 can provide acommunication interface to various external devices. The display node100A can be coupled to a network 160 that comprises one or more of alocal area network (LAN), a wide area network (WAN), a wireless localarea network (WLAN), a personal area network (PAN), a virtual privatenetwork (VPN), or the Internet, for example, via a wired, wireless, orcombination of wired and wireless, communication link 115. The network160 communicates with various computing devices and/or other electronicdevices via wired and/or wireless communication links. In someembodiments, the network 160 comprises an IEEE 802.11g WLAN.

In some embodiments, the display node 100A can include, or may becoupled to via a network connection, a processed image data source 162,such as a database, that includes information about one or more imagesto display on the tiled array display. The information supplied by theprocessed image data source 162 can include a full size or originalimage that was or will be preprocessed and stored in a hierarchicalformat that includes sub-images, with each sub-image being a reducedsize and/or reduced resolution version of the original image. Forexample, a reduced resolution sub-image can be generated from anoriginal full resolution image by deleting rows and columns of thepixels of the original image at predetermined spacings, therebygenerating a lower resolution version of the full image. A reduced sizesub image can be generated by cropping the original image. Any othertechnique of creating sub-images can also be used.

In some embodiments, the largest sub-image can be the same size as theoriginal image and/or include image content from the original image. Forexample, each sub-image can be stored as one or more blocks, or tiles,to allow rapid access to a particular part of the original image withouthaving to access entire rows and/or columns of pixels. In someembodiments, this can allow the display node 100A to fetch exactly thelevel of detail (sub-image) it requires and/or to quickly fetch theneeded blocks that make up the portion of the image to be output to thedisplay 166. In addition to the devices that are illustrated in FIG. 1B,the display node 100A can be connected to other computing devicesthrough a bus or the network 160.

In some embodiments, the original image data source 164 can include oneor more original or full size digital images. The digital images cancomprise JPEG, GIF, PNG, TIFF, BMP, and/or other digital image formats.In other embodiments, the original image data source 164 can include oneor more original or full size digital images that can be tens orhundreds of millions of pixels (e.g., 200-600 Megapixels), or evenbillions of pixels. In some embodiments, the display node 100A canpreload and preprocess the original images stored in the original imagedata source 164 and store the result in a hierarchical format in theprocessed image data source 162. In other embodiments, the display node100A can determine the correct portion(s) of the original image(s) to bedisplayed on its associated display 166 and output the correspondingpreprocessed image data for display. Thus, the processed image datasource 162 can be used to reduce the amount of data that needs to beloaded in memory and support faster manipulation or modulation of imageson the tiled array display.

In some embodiments, the hierarchical format used for storing theprocessed image data comprises a tiled pyramidal TIFF format. However,other formats can also be used.

The tiled pyramidal TIFF format can allow the display node 100A to storemultiple resolutions of a preloaded image in the mass storage device120, memory 130 or in the processed image data source 162. In addition,the tiled pyramidal TIFF format can allow the display node 100A topartition the original image into smaller tiles of data so as to reducethe amount of image data to be fetched and processed, thereby enhancingsystem performance.

In some embodiments, the original images stored in the original imagedata source 164 can be compressed or uncompressed images. In someembodiments, the processed image data source 162 can also be configuredto receive a compressed image from the original image data source 164.Once received, the display node 100A can decompress an original imageand then preprocess the original image into a set of one or more imagesthat are compressed or decompressed and store them in the processedimage data source 162, in the mass storage device 120, or in memory 130.In some embodiments, each tile of the tiled pyramidal TIFF images can becompressed using lossless or lossy compression algorithms, such asDeflate, LZW, or JPEG. Spatial identifiers can be used to identifyvarious portions or resolutions of the sub-images to facilitateefficient extraction of different regions or resolutions of the originalimage(s).

In some embodiments, one or more of the image data sources may beimplemented using a relational database, such as Sybase, Oracle,CodeBase and Microsoft® SQL Server, as well as other types of databasessuch as, for example, a flat file database, an entity-relationshipdatabase, an object-oriented database, and/or a record-based database.

With continued reference to FIG. 1B, in some embodiments the displaynode 100A can also include application modules that can be executed bythe CPU 105. In some embodiments, the application modules include theimage processing module 150 and the image display module 155, which arediscussed in further detail below. These modules can include, by way ofexample, components, such as software components, object-orientedsoftware components, class components and task components, processes,functions, attributes, procedures, subroutines, segments of programcode, drivers, firmware, microcode, circuitry, data, databases, datastructures, tables, arrays, and variables.

In some of the embodiments described herein, each display node 100A canbe configured to execute instructions in the image processing module150, among others, in order to support user interactivity by reducingthe amount of data loaded into the memory 130 when an image needs to bedisplayed on the interactive tiled display system 50. In addition, theimage processing module 150 can allow portions of several images to beresident on the display 166, thus supporting display and manipulation ofmultiple large images across the multiple display nodes 100. Forexample, in some embodiments, an original large image can be tens ofbillions of pixels. The image processing module 150 can preprocess andstore multiple full size or original images by determining the correctportion of the original images to be displayed on a specific displaynode 100.

In some embodiments, each original image can be stored in a hierarchicalformat (e.g., tiled pyramidal TIFF format) that includes sub-images thatcan be a reduced size and/or reduced resolution version of the originalimage. In some embodiments, the largest sub-image can be the same sizeas the original image and/or include image content from the originalimage to support resizing of the image, such as zooming in and/or out.The image processing module 150 can store each sub-image of the originalimage as one or more blocks, or tiles, to allow rapid access to aparticular part of the full size image without having to access entirerows or columns of pixels. This advantageously allows a display node100A that knows which portion of the original image is needed to outputon its display 166 to fetch the level of detail needed, such as asub-image and/or to quickly fetch the needed blocks that make up theimage portion. Such blocks or tiles can be provided with identifiersthat indicate the position of the bock or tile in the original image,and/or other characteristics. The image processing module 150 can beconfigured to store each sub-image of the original image as one or morelayers of resolution to allow rapid access to a particular resolutionlayer without having to decompress or compress large resolution imagesfor display, thereby improving efficiency. Storing multiple resolutionsof the preloaded image can advantageously allow the use of parallelprocessing techniques (e.g., threading) to improve performance speed.

The image processing module 150 can be further configured to sendrequests to control node 102 for information about other display nodes(e.g., 100B, 100C, etc.) and/or vice versa. In some embodiments,messages can be exchanged between the control node 102 and/or otherdisplay nodes (e.g., 100B, 100C, etc.) that include information aboutthe overall state of the aggregate tiled display, or about a particulardisplay node. In some embodiments, the messages comprise ScalableParallel and Distributed Systems (SPDS) messages. SPDS Messaging is aC++ library that encompasses sockets and data into abstractions calledEndpoints and Messages. In some embodiments, the control path usesmulticast UDP Endpoints, which means Messages are received by one ormore subscriber nodes. Broadcast UDP Endpoints can also be used to sendMessages to all nodes on a subnet. The SPDS Messaging library can useUnix/Winsock socket operations to implement the Endpoints and the sendand receive operations used to transfer Messages.

The display node 100A can also execute instructions in the image displaymodule 155 to display one or more images or portions thereof andmanipulate the images. As noted above, an original image that is fullsize can be preprocessed by the image processing module 150 and thenstored in the processed image data source 162 or in other local memory(e.g., memory 130). Because the amount of data loaded into memory 130can be reduced when an original image is stored in a hierarchical format(such as tiled pyramidal TIFF), the image display module 155 can enablea highly interactive display space that spans multiple display nodes100. By storing the data in a hierarchical format, the tiled displaysystem 50 can implement further parallel processing techniques (e.g.,MIMD or SPMD techniques and/or multithreading) to improve systemperformance.

For example, the image display module 155 can load the appropriatesub-image of an original image in memory 130 and on the display 166. Insome embodiments, surrounding blocks and blocks from higher and lowerlevels can also be pre-fetched for higher performance by the imagedisplay module 155. This can allow each display node 100A to support thedisplay of more than one such image or portions thereof. For example,the image display module 155 can determine its individual displayboundary based on the resolution layer determined by the imageprocessing module 150 (e.g., the resolution layer having the smallestimage size larger than the native resolution of the display 166). Basedon the determination of the individual display boundary, the imagedisplay module 155 can determine which tiles overlap with its displayboundary and can fetch the overlapping and/or surrounding blocks, ortiles. Additionally, a resource management approach can supportinteractivity by reducing the amount of data loaded and allowingportions of several images to be resident on each tile, thus supportingdisplay and manipulation of multiple large images.

Advantageously, the image display module 155 can be configured to allowthe use of multiple highly detailed images, which can exceed billions ofpixels, to be displayed as part of a high resolution, coordinatedworkspace on a tiled display that includes multiple display nodes 100.Further, the image display module 155 can be configured to allow inreal-time or in near real-time interaction with multiple images byallowing moving, zooming, rotating, color filtering, and transparencycontrolling of images on the display node 100. In some embodiments, theimage display module 155 can be configured to perform color filtering,shading or transparency control using a vertex and/or pixel shader, suchas a Cg shader.

For example, in some embodiments, the user may use a front endinterface, such as the control node 102, and select to rotate an imageon the tiled display. In some embodiments, the user can manipulate thedisplay via user input devices 151 (for example, a keyboard or 3Dmouse). The image display module 155 can respond to the user'sselection, by using a reduced size or reduced resolution version (e.g.,thumbnail) of the original image, which may be stored in the processedimage data source 162, to quickly adjust its display 166. For example,when the image on the system is initially selected for rotation, theimage display module 155 can replace the image being displayed with thethumbnail during the rotation process. In some embodiments, as thethumbnail of the original image is rotated and thus redrawn at differentangular orientations, less processing power is required to complete theredraw process, thereby providing a quicker response time. In someembodiments, an increased resolution image can be displayed uponcompletion of the rotation or other manipulation of the image.Additionally, user manipulations such as those noted above, can be usedto generate messages sent to the control nodes, described in greaterdetail below.

In addition, the image display module 155 can also be configured toexchange messages with the control node 102 or the other display nodes(e.g., 100B, 100C, etc.) about the state of the tiled display, such aswhich portion of the original image needs to be displayed by respectivenodes. Thus, the image display module 155 can provide a highlyinteractive experience that has numerous applications, including themanipulation of data about medical conditions, cancer cells, satelliteimages, geosciences, oil monitoring, weather monitoring or prediction,astronomy, and the like.

Although FIG. 1B has been described with respect to display nodes 100, acontrol node 102, and an image data source 164, certain of the featuresof the system shown in FIG. 1B can be implemented using other types ofcomputing devices communicating over the network 160. For example, thecontrol node 102 can communicate over the network 160 with a mediasource device (instead of the image data source 164) and one or moredestination computing devices (instead of the display nodes 100). Thecontrol node 102 can broker a connection between the media source deviceand a destination computing device. In some embodiments, the controlnode 102 locates media data stored on the media source device andobtains the media data or a portion thereof (such as a thumbnail) fromthe media source device. The control node 102 can also be configured tosend the media data or the portion thereof to the destination computingdevice, along with network communication or connectivity data. Thenetwork communication data can enable the destination computing deviceto communicate with the media source device to obtain media data. Thenetwork communication data could include, for example, a network address(such as an IP address) of the media source device, a proxy for themedia source device, an anycast IP address for a plurality of mediasource devices, or the like.

In some embodiments, providing the network communication data from thecontrol node 102 to the destination computing device enables thedestination computing device to obtain media, including media updates,from the media source device. As a result, the control node 102 can beless of a bottleneck for communications between the media source deviceand the destination computing device.

In some embodiments, the destination computing device can report orotherwise provide the media updates it receives or a portion thereof tothe control node 102. For example, the destination computing device canprovide a thumbnail, a reduced frame rate video, metadata associatedwith the media updates, combinations of the same, and the like. Thecontrol node 102 can therefore keep track of the media data provided tothe destination control device.

In some embodiments, the control node 102 can provide networkcommunication information to the media source device instead of or inaddition to providing communication information to the destinationcomputing device. This network communication information can allow themedia source device to communicate with the destination computingdevice. For example, the control node 102 can provide a network addressof the destination computing device to the media source device. Themedia source device can then push media to the destination computingdevice.

In some embodiments, the control node 102 can identify media stored onthe media computing device without requesting the media. The controlnode 102 can provide network communication data to the destinationcomputing device, which allows the destination computing device toobtain the media from the media server. Thus, little or no media mightpass through the control node 102 from the media source device to thedestination computing device, further reducing bottleneck effects of thecontrol node 102.

In some embodiments, the interactive tiled display system 50 can also beconfigured to support other data formats, in addition to large digitalimages. For example, the interactive tiled display system 50 can beconfigured to support and display Normalized Difference Vegetation Index(NDVI) data, MRI scan data, SOAR 3D terrain data, digital video data(including standard and HDTV format), and streaming content (e.g., froma webcam).

The interactive tiled display system 50 can also be configured todisplay a real-time replica of content displayed on the display screenof one or more computing devices in communication with the network 160,as described in U.S. patent application Ser. No. 12/487,590 entitled“Systems, Methods, and Devices for Dynamic Management of Data StreamsUpdating Displays,” the entire content of which is hereby expresslyincorporated herein by reference in its entirety. In some embodiments,the interactive tiled display system 50 comprises middleware that can beconfigured to render and display an OpenGL application by executinginstructions to launch and manage instances of the OpenGL application oneach of the display nodes 100 in parallel through a thread-based networkcommunication layer.

FIG. 2 is a flowchart illustrating an embodiment of a method ofpreprocessing images that can provide a high level of interaction andmanipulation of the images on tiled display systems. The methodillustrated in FIG. 2, as well as other methods disclosed below, can bestored as process instructions (for example, on any type ofcomputer-readable storage medium) accessible and executable by the imageprocessing module 150 and/or other components of the display node 100A,the control node 102, or any other computer or system connected to thetiled display system 50 directly or over any type of network. Dependingon the embodiment, certain of the blocks described below can be removed,others may be added, and the sequence of the blocks can be altered. Insome embodiments, each of the display nodes 100 can be configured toperform the method, as well as other methods discussed below, usingparallel processing techniques to improve system performance.

Beginning at block 210, one or more full size or “original” images isreceived by each of the display nodes 100. In some embodiments, the oneor more full size images can be sent from the original image data source164 to the display nodes 100 by way of the control node 102. The fullsize images can also be sent over the network 160 from a computingdevice or received via a physical I/O port of the display node 100A,such as a USB port.

The full size images can include various types of data forvisualization, such as still photographs, videos, or other images ofanything including but without limitation, medical, satellite,geosciences, oil, weather monitoring or prediction, astronomy imagery,and include those discussed above with reference to FIG. 1B. As those ofskill in the art will recognize, the image types can vary greatly and bebased on a variety of possible types of data. After the full size imagesare received in operation block 210, the process can move to block 220.

In block 220, a set of sub-images that allow for rapid access toportions of the one or more full size images can be created. In someembodiments, the image processing module 150 preprocesses each originalfull size image and creates a set of sub-images that are a reduced sizeand/or reduced resolution (e.g., thumbnail) version of the originalimage. For example, a sub-image can be formed from a cropped portion ofthe original image, at the same resolution of the original image. Forexample, a 10240×7680 pixel image can be stored as 256×256 pixel tiles.In such an embodiment, the overall image comprises 40×30 tiles, orsub-images, each comprising 256×256 pixels. Other tile sizes can also beused. In some embodiments, the tiles can be roughly square. In otherembodiments, the tiles can be comprise rectangular strips. Storing theimage data as sub-images of smaller tile sizes can reduce the totalamount of data to be processed for display on each of the display nodes100.

Each of the display nodes 100 can be configured to only decompress orotherwise “process” the tiles necessary to cover its display area ratherthan decompressing or processing the entire image. For example, each ofthe display nodes 100 can be configured to process groups of the tilesthat correspond to the respective portions of the image to be displayedon each display unit. In operation, each of the groups can comprisedifferent subsets of all the tiles, although not necessarily exclusivegroups. For example, groups associated with two adjacent display unitsmay have one or more tiles in common.

Storing the image data in a tiled format can advantageously allow fasterprocessing when the zoom level of the image is increased to show moreprecise detail. For example, instead of processing the lower zoom levelportion of the image displayed on a particular display node, the displaynode can be configured to process only the tiles corresponding to thehigher zoom level portion of the image. In some embodiments, if aparticular tile overlaps a boundary between adjacent display nodes, eachof the display nodes containing a portion of the image on theoverlapping tile can process the same overlapping tile.

In some embodiments, the sub-images can comprise multiple resolutionlayers of the original full size image. A first sub-image can be createdthat is the same resolution as the original image, a second sub-imagecan be created that is half the resolution of the first sub-image, athird sub-image can be created that is half the resolution of the secondsub-image, and so on. For example, a 10240×7680 pixel image can bestored in five resolution layers as follows:

-   -   Layer 0: 10240×7680    -   Layer 1: 5120×3840    -   Layer 2: 2560×1920    -   Layer 3: 1280×960    -   Layer 4: 640×480

In some embodiments, the resolution layers can also be processed intotiles or tiled sub-images. Storing the image in multiple layers ofresolution can advantageously improve efficiency. For example, if theresolution of the display 166 is much smaller than the resolution of theimage, the display 166 can only show the detail at the lower resolutionand actual pixel details of the image are lost. In some embodiments, itcan be advantageous to decompress a lower resolution layer of the imagethan the original high resolution image.

In some embodiments, the image processing module 150 can createsub-images of neighboring portions of the large image. For example, theimage processing module 150 can create sub-images of portions of thelarge image to be displayed on neighboring display nodes. This canadvantageously result in improved processing efficiency when the imageis manipulated (e.g., rotated, panned, or resized) by a user and one ofthe sub-images overlaps a border between two adjacent display nodes.After the operation block 220, the process can move to block 230

In block 230, the set of sub-images can be stored in the processed imagedata structure 162 or in memory. In some embodiments, each sub-image canbe stored in a hierarchical format (e.g., tiled pyramidal TIFF), suchthat one or more blocks, or tiles, allow rapid access to a particularpart of the image without having to access entire rows or columns. Insome embodiments, this can advantageously allow a display node 100A ofthe tiled system that knows the portion of an image it needs to displayto fetch exactly the level of detail, for example a correspondingsub-image, and/or to quickly fetch the needed tiles that make up theimage portion to be displayed. The sub-images can include a spatialand/or resolution identifier to allow for quick retrieval from theprocessed image data structure 162. In some embodiments, the sub-images(e.g., tiles) can be identified and retrieved based on a positioncommand, coordinate, or value received in a message from the controlnode 102 (e.g., a “requested position”). For example, if the size ofeach tile is 256×256, a position coordinate of (0,0) or (124,0) canreturn the first tile and a position coordinate of (256,0) or (300,0)can return the next tile down on the x-axis. The identification andretrieval of tiles can be an operation supported by the TIFF softwarelibrary (libTIFF).

In some embodiments, the image data (which can be divided intosub-images of various resolution layers and tiles) is stored as a singlefile format, with the sub-images being capable of being retrieved by theTIFF software library. In other embodiments, each of the sub-images canbe stored individually, for example, in a directory file system. Forexample, each resolution layer can be a different level of directory ina file system, with each directory including all of the tiles for thatresolution layer.

FIG. 3 schematically illustrates an embodiment of another method ofdisplaying and manipulating images on a tiled array display system.Depending on the embodiment, certain of the blocks described below maybe removed, others may be added, and the sequence of the blocks may bealtered. In some embodiments, each of the display nodes 100 can beconfigured to perform the method in parallel, thereby resulting infaster display and movement of the overall image.

Beginning at block 310, a portion of a full size image to display on aparticular display node (e.g., display node 100A) can be calculated ordetermined. Additionally, multiple portions of one or more full sizeimages to display can also be calculated or determined. Advantageously,this reduces the amount of data to be loaded on each particular displaynode 100, as well as a controlling computer, such as the control node102, and thus increases the responsiveness of the overall tiled arraydisplay. Because of the increased responsiveness, manipulation of imageson the tiled array display can be improved.

Moving to block 320, the one or more sub-images that correspond to theportion of the full size image to display are loaded into memory 130,for example. Because disk access times for loading a full size image canbe impractical, each display node 100A can load tiles or blocks of theappropriate sub-images needed for its local portion of the overall tileddisplay. In some embodiments, the correct portions of multiple full sizeimages can also be loaded into the memory 130 of the correspondingdisplay node 100A, 100B, 100C, etc. The loading of tiles canadvantageously be performed using parallel processing techniques, suchas multithreading. Multiple threads can be used to simultaneously loadthe requested tiles from the processed image data source 162 or from themass storage device 120 or other memory/file system.

Moving to block 330, the one or more sub-images are displayed. In someembodiments, the display node 100A may render the sub-images resident inthe memory 130 using a multimedia device 140, such as a video card. Therendered sub-images can then be placed on display 166, such as an LCDmonitor.

FIG. 4 schematically illustrates an embodiment of a method ofcontrolling the display of one or more large images (e.g., hundreds ofmegapixels or larger) on an interactive tiled array display. Dependingon the embodiment, certain of the blocks described below may be removed,others may be added, and the sequence of the blocks may be altered.Although the method is described with reference to a single display node100A, the method can be performed by any of the display nodes 100. Insome embodiments, each of the display nodes 100 can be configured toperform the method in parallel, thereby resulting in faster display andmovement of the overall image.

At Block 405, the display node 100A receives a message (e.g., a command,data, packet, etc) from the control node 102, or another source, via thenetwork 160. The message can be sent over the network 160 by any numberof communication mechanisms, for example, TCP/IP over Ethernet orspecialized high-speed interconnects, such as Myrinet. In someembodiments, the message contains information regarding an image to bedisplayed, or an image that is currently displayed, on the tiled arraydisplay.

For example, the message can comprise initial display parametersdesignated for an image that has not yet been displayed or an image thatis currently being displayed. Such display parameters can comprise dataindicating a desired or requested position for the corresponding image.Hereinafter, the content in the message indicative of the “position”included in the message is referred to as a “requested position.”

Such a “requested position” could be generated by the control node 102when a user places, drags to, or clicks on a representation of a subjectimage (510 in FIG. 6) on a wire frame representation of the array 10 ona user interface 600 (FIG. 6). In such an example, the user interface600 can calculate, estimate, etc., the location of a reference pixel inthe image 510 on the array 10.

The reference pixel of the image 510 can be any pixel of the image. Insome embodiments, the reference pixel can be the center pixel of theimage, or a pixel immediately above, below, left or right of the centerof the image, if the image does not mathematically have a center pixel.Other reference pixels or reference parts of the image can also be used.

The portion of the message indicative of the requested position can bein any form. For example, the “position” can be expressed as a row andcolumn of a virtual pixel grid representing all the pixels of the entirearray 10. In some embodiments, the requested position can be indicativeof a pixel on a particular display unit, e.g., pixel 1, 1 on displayunit 100G. Other formats can also be used.

The message can also include other aspects of the requested position,such as angular orientation. For example, the message can include dataindicative that the image should be rotated by an angle. The angle canbe relative to the reference pixel, such as the center pixel notedabove, any other pixel of the image, or any other point of reference.

The message can include further aspects of the requested position, suchas zoom or magnification. For example, the message can include anindication that the image should be presented in a 1 to 1 manner, e.g.,every pixel of the image represented on a respective single pixel of thecorresponding display units. The message can also include other dataindicative of other display parameters.

In some embodiments, the message can contain image data, for example,streaming video data, still image data, or a live display feed from acomputing device or other data source in communication with the displaynodes 100 via the network 160 as described in U.S. patent applicationSer. No. 12/487,590 entitled “Systems, Methods, and Devices for DynamicManagement of Data Streams Updating Displays,” the entire content ofwhich is hereby expressly incorporated herein by reference in itsentirety.

With continued reference to Block 405, the message can be received viabroadcast, multicast, unicast, a reliable network overlay topology toparallelize transfer, or any other network messaging technique. In someembodiments, the multicast messages can be transmitted by unreliablemulticast with error checking and handling. In some embodiments, thedisplay nodes 100 can subscribe to a distribution group. For example,the display nodes 100 involved in displaying a portion of an image thatdoes not cover the entire tiled display array can subscribe to, or join,a multicast group address. After the display nodes 100 have joined amulticast group, they can receive the multicast messages sent to themulticast group address.

In some embodiments, the message can comprise a configuration message toinitialize or reconfigure a coordinate system of the tiled arraydisplay. The configuration message can include a node identifier and anode coordinate. In some embodiments, the coordinate system can bedetermined using letters for the columns and numbers for the rows inspreadsheet fashion. For example, the top left display node can be A1,the display node just to the right of A1 can be B1, the display nodejust below A1 can be A2, and so on.

In some embodiments, the coordinate system can be based on a numericalmethod. For example, the top left display node can be (0,0) with amonitor size of (1.0, 1.0). The display node beneath the top leftdisplay node can be (0,1) with a monitor size of (1.0, 1.0). In someembodiments, different monitor sizes can be applied when using thenumerical coordinate system. Such a coordinate system, or othercoordinate systems can be used to allow each of the display nodes 100 todetermine which portion of an image to display on its associated display166, as also described above.

In other embodiments, the display nodes 100 can be “pre-configured.” Forexample, the control node 102 can remotely access each of the displaynodes 100 (for example, using virtual network computing (VNC) software)and rename, or otherwise configure, the display nodes 100, according toa predefined display configuration scheme (for example, the coordinatesystems described above).

In some embodiments, the message received from the control node 102 canalso comprise a state message. State messages can be received atperiodic intervals or any time a change in a state of an image occurs(for example, due to user interaction or manipulation of the image onthe user interface 600, FIG. 6). The state message can be received viabroadcast to all of the nodes, multicast to a subset of the nodes, orunicast to a single node.

The state messages can include state values of the image to be displayedor updated state values of the image currently being displayed. Thestate values can include values regarding image position, image size,image resolution, image cropping, angle of rotation, pixel colorimetryor shading, transparency and/or other display parameters. For example,as described above with regard to messages generally, the state valuescan include X and Y values to indicate a position of the image. In someembodiments, the X and Y values can indicate a delta change in position,or offset, from the previous position. Received X and Y values can alsoindicate an updated location of the central pixel or another referencepixel of an image. In some embodiments, the X value indicates the leftboundary of the image and the Y value indicates the top boundary of theimage. The X and Y values can be represented in resolution units orcoordinate values, for example.

Additionally, X and Y values can be used to indicate a size of the imageon the overall tiled display. For example, the message can include twosets of X and Y values, indicating the locations of two reference pixelsin the image. In an example, the two sets of X and Y values can indicatethe locations of two opposite corners of the image. However, referencepositions of any two reference pixels in the image could also be used.The X and Y values can be updated to accommodate panning of the imagearound the tiled array display.

The state values can include a Z value to indicate a zoom level or“magnification” of the image. The Z value can be updated to accommodateresizing of the image by zooming in and out. In some embodiments, the Zvalue can be used to determine the appropriate resolution layer toaccess as a starting point for processing of the image, as described inmore detail above. For example, the resolution level just larger (inboth height and width) than, or equivalent to, the specified zoom levelcan be selected as the starting point. As noted above, the state valuescan also include an angle value. The angle value can be used to indicatea requested angle of rotation of the corresponding image. The anglevalue can be updated to accommodate rotation of the image on the tiledarray display.

In some embodiments, the state values can include colorimetry values ofthe image pixels. In some embodiments, the colorimetry values caninclude red, green, and blue (RGB) color values. In other embodiments,other color schemes, such as CMYK, can be used. The colorimetery valuescan also include information configured to adjust other parameters orviewing conditions of the image or to transfer from one color scheme toanother, such as chromacities or transfer functions. The colorimetryvalues can be updated to accommodate color filtration. Color filtrationcan advantageously be used to expose new visual information or to matchdesired aesthetics (for example, for medical image data). The statevalues can also include an alpha value to indicate a level oftransparency of the pixels. The alpha value can be used to accommodateadjustment of the transparency of the image (from fully opaque to fullytransparent) and to accommodate visual overlays of multiple images.

When multiple images are being displayed on the tiled array display andstate messages are being sent via broadcast, multiple state messages canbe transmitted, with each broadcast including an ID of the image towhich it pertains in addition to the state values of the image andmultiple state messages can be sent. Other image values can also beincluded within the messages sent by the control node 102.

In some embodiments, as noted above, the message can include an imageportion and a display parameter portion. The image portion can compriseimage data, such as a computer file in the form of a compressed oruncompressed image, or an identification of an image that is alreadystored on local memory (e.g., 130 FIG. 1B) of the display units. Thedisplay parameter portion of the message can comprise commands orinstructions regarding the display and rendering of the large digitalimage on the tiled array display. For example, the display parameterportion can include instructions indicating where the image is to bepositioned on the tiled array display if it is not intended to take upthe entire screen and/or at what resolution the image is to bedisplayed.

In some embodiments, the display parameter portion can includeresolution information to indicate the resolution with which to displaythe large image. In other embodiments, the display nodes 100 candetermine the appropriate resolution based on the native resolution oftheir associated displays 166 and the various levels of resolution ofthe received large image. In some embodiments, the display parameterportion can include colorimetery or transparency information.

At decision block 410, the CPU 105 of the display node 100A determineswhether the state message impacts a display of an image or other data onits associated display 166. If no action is required by the display node100A, then the display node 100A continues to receive the statemessages. If, in decision block 410, it is determined that action isrequired, then the process continues to block 415.

For example, each of the display nodes can include a position moduleconfigured to store data indicative of any position in a tiled arraydisplay. For example, such a module can be anywhere in the memorysystem. Further, because there is no clear limit on the possible size ofthe array 10, the module can be configured to accept and retain an inputindicative of any position on an array of any size. In some embodiments,the module can be embodied in software and accessible through a userinterface, not illustrated.

In preparing a display node 100A for operation as a node in an arrayeddisplay, such as the array 10, the display node 100A can be physicallypositioned in the array. For example, the display node 100A can bepositioned adjacent other nodes in an array, such as the array 10. Auser can also input at least a first data indicative of the position ofthe display node 100A in the array, the position module receiving andretaining the at least a first data or other data indicative of theposition. For example, the user can remotely connect to the display node100A through the use of VNC software, as described above, to indicatethe position of the display node 100A in the array.

Each of the display nodes can also include one or more an imageprocessing modules, such as the image processing module 150, which canbe combined into a single module with the position module describedabove, or other modules described herein, can be separate from all othermodules, or combined into any of various possible combinations of all ofthe modules described herein. In some embodiments, the module used forthe determination performed in decision block 410 can be referred to asan “action required determination module.”

In some embodiments, the image processing module 150 can be configuredto process the message received in operation block 405 in order todetermine if the information in the message is indicative of a requestfor the display unit to process and generate an image on itscorresponding local display 106. In some embodiments, the display nodes100 can generate a virtual map of the entire array 10, and calculate theboundaries of the image resulting from processing the image in themanner indicated in the message. In some embodiments, the resultingboundaries can be compared to the position of the node 100A retained inthe position module, described above.

If the position of the resulting boundaries indicates that no part ofthe resulting image would lie on the display unit at the position in thearray retained in the position module, then the result of decision block410 is “NO”, and the routine can return to operation block 405 andrepeat. If, on the other hand, the result indicates that any part of theresulting image would lie on the display unit at the position in thearray retained in the position module, then the result of decision block405 is “YES” and the process can continue to operation block 415.

The display units can also be configured to perform other analyses onthe information in the message. For example, if the message includesinformation indicating that only a portion of an image currentlydisplayed on the display unit has changed, but the portion of the imagedisplayed on the display unit performing the analysis has not changed,then the result determined in decision block 410 can also be “NO.”

At block 415, each of the display nodes 100 can determine the respectiveportion of the large image that it is responsible for displaying basedon the state values of the message. In some embodiments, each of thedisplay nodes include a module for further processing the messagereceived to determine what discrete part of the image is to be displayedon the display node performing operation block 415. For example, in acontinuation of the operation described above with reference to decisionblock 410, or repeating all of those processes, the display node candetermine which pixels of the image fall within its position in theabove described virtual array.

In some embodiments, where the image file is stored on the local displaynode memory in a compressed format, the display node can decompress theentire image or only a discrete part of the image, depending on thecompression format. Optionally, the display node can process the imagefile at the maximum resolution of the image, or at another resolution,for example, another resolution included in a pyramidal image dataformat.

In some embodiments, the display node can first calculate a mosaicrepresentation of the image, using the boundaries of the sub tiles ofthe image noted above. In this example, the display unit can perform thedetermination of operation block 415 by determining which tiles or subtiles of the image would be positioned on the display unit, if the imagewas oriented according to the message. In some embodiments, the displayunit can be configured to determine that a sub tile would be positionedon the display unit if any portion of the sub tile would lie on thedisplay unit if the image were positioned according to the message.

A determination, such as the above-described determination performed inoperation block 415, can be performed by an image processing module,such as the image processing module 150, an additional module combinedwith the image processing module 150, a separate image processingmodule, or any other device or module configured to perform thefunctions described herein. The module used to perform the operationperformed in operation block 415, can be referred to as a “portionidentification module.”

After determining the discrete portion of the image to display whichcould include the entire image, a specific list of pixels, a range ofrows and columns of pixels, a list of sub tiles of the image, or anotherdescription of the discrete portion, the routine can continue tooptional operation block 420.

At optional block 420, the display node 100A can update image data fordisplay, if beneficial, for example, based on the state values of themessage or based on the specific techniques used to perform the abovesteps. For example, if the Z value has changed (e.g., indicating a zoomin has been requested), new tiles or sub-images can be loaded toillustrate the increased detail of the image portion being displayed onthe display node 100A. In some arrangements where new image data isreceived, the display node 100A can create one or more sub-images of thelarge image based on the image data corresponding to the portion of thelarge image that it is responsible for displaying.

In some embodiments, for example, where the display units perform theoperation block 415 based on the position of tiles or sub tiles of theimage, the display unit can process the tiles or sub tiles identified inoperation block 415. As noted above, depending on information in themessage, such as an indication of magnification, zoom, or otherrequested parameters regarding the display of the image, the displayunit can selectively process the image data according to the identifiedtiles or sub tiles of a selected resolution or layer of the originalimage data, for example, where a the original image is in the form of apyramidal resolution format image, with each of the differentresolutions broken down into tiles or sub tiles. This technique canprovide a further advantage in that, if an image spans at least two (2)display units such that two different portions are identified by theserespective display units in operation block 415, neither of the displayunits need to process the entire image. Rather, the analysis ofoperation block 415 can be used to help the display units to efficientlydetermine which part of an image to process for display, without havingto process, resolve, decompress, map, etc., the entire original image atfull resolution.

One or more sub-images can be generated by the image processing module150 as described above with respect to block 220 of FIG. 2. The createdsub-images can be stored in memory or in a preprocessed image datastructure, as described above with respect to block 230 of FIG. 2. Thestep of updating data in operation block 420 can include loadingindividual tiles, or sub-images via multithreading. For example,multiple tile threads can be configured to run in parallel to increaseperformance speed, as described above.

At block 425, the display node 100A outputs its respective portion ofthe large image on its associated display 166, thereby presenting ahigh-resolution rendering of the original large image on the tiled arraydisplay. When an image is being moved around the display quickly,certain tiles may not be loaded from a tile thread in time to display onthe display 166, resulting in black tiles being displayed. Accordingly,in some embodiments, a lower or the lowest resolution layer image can beused, processed, displayed, etc., to temporarily mask the black tilesuntil the higher resolution tile is loaded from the tile thread.

In some embodiments, such as when an image spans multiple display nodes,the display nodes 100 can be configured to adjust their respectivedisplay output to compensate for the presence of bezels between thedisplay nodes 100. For example, the display units can be configured tooperate in two or more modes, such as an aspect ratio preservation modeor an image data preservation mode.

In an “aspect ratio preservation mode”, in the decision block 410 andthe operation block 415, the display units can use a map of all thepixels in the array 10 which includes virtual pixels that fill the gapsbetween adjacent display nodes. These virtual pixels are not assigned toany of the display nodes. As such, an image displayed on the array 10which overlaps multiple display units will have portions of the imagemissing, i.e., any portion of a displayed image falling within the gapsformed by the bezels, would not be displayed. However, the portions ofthe image that are displayed will have the aspect ratio of the originalimage. As used herein, the term “unified display of the digital image onthe array of display units” includes an image displayed as in the aspectratio preservation mode, despite the missing data.

This mode can be desirable when viewing images for aesthetics, images orart, architecture, etc., for example, because the alignment of featureswhich span bezels will be preserved, as well as other applications.However, the image data preservation mode, described below, can also bedesirable for viewing images for aesthetic reasons.

In an “image data preservation mode”, in the decision block 410 and theoperation block 415, the display units can use a map of all the pixelsin the array 10 which does not include or utilize unassigned pixels inthe gaps between adjacent display nodes. Instead, every pixel of theoriginal image to be displayed on the array 10 is assigned to a displaynode. As such, the aspect ratio of an image displayed on the array 10which overlaps multiple display units will be affected/distorted, in theportion spanning the bezels between the displays (166 FIG. 1B) of thedisplay nodes. However, the portions of the image that are displayedwill have the aspect ratio of the original image. As used herein, theterm “unified display of the digital image on the array of displayunits” includes an image displayed as in the image data preservationmode, despite the distortion on the resulting aspect ratio.

This mode can be desirable for scientific analyses of images because allof the image data is displayed, regardless if an image spans a bezel.However, the aspect ratio preservation mode can also be helpful inscientific analyses, for example, where it is desired to compareproportional sizes of features of one or more images.

In some embodiments, the display nodes 100 can be configured to activateone or a plurality of predetermined functions to change the positionand/or size of the image to conform to one or more physical or logicalboundaries of the tiled array display, such as the “snapping” functiondescribed in U.S. Provisional Application 61/218,378, entitled “Systems,Methods, and Devices For Manipulation of Images on Tiled Displays,” theentire content of which is hereby expressly incorporated herein byreference.

FIGS. 5A and 5B schematically depict how an image can be displayed onone or more display units 100A, 100B, 100C, 100D of a tiled array 500.The tiled array 500 can be in the same form as the array 10, or anotherform.

As depicted in FIG. 5A, an image 510 can be displayed in such a mannerthat it overlaps two display units 100A and 100C of the array 500. Asfurther detailed in FIG. 5B, if the array 500 is only displaying animage on a limited number of the total number of display units, such asonly the units 100A and 100C, then the associated image data from theimage source can be selectively sent to those display units 100A and100C (e.g., via a multicast message). This can help reduce bandwidthrequirements for transmission of image data.

For example, with continued reference to FIGS. 5A and 5B, the image 510,when overlapping two display units 100A, 100C, can be broken down intoparts corresponding to those portions displayed on different displayunits. As shown in FIG. 5B, the image 510 can be segregated into a firstportion 511 (displayed on display unit 100A) and a second portion 512(displayed on display unit 100C). As such, in some embodiments, thetiled array 500 can be configured to send the image data correspondingto the portion 511 to the display unit 100A and the other image datacorresponding to the portion 512 to the display unit 100C, along withdata indicating the position at which these portions 511, 512 should bedisplayed.

As such, the image data corresponding to the image 510 does not need tobe broadcast to the other display nodes (e.g., 100B, 100D). Rather, insome embodiments, the control node 102 can be configured to only sendimage data to those display nodes in the array 500 that will be involvedin displaying at least a portion of the image 510 (e.g., via multicast).This can greatly reduce the magnitude of data flowing into and out ofeach of the display nodes 100.

With reference to FIG. 6, in some embodiments, the control node 102 canbe configured to generate a schematic representation of the displaynodes 100 for use as a graphical user interface. In some embodiments,the control node 102 can be configured to display the images beingdisplayed on the tiled array display on its own associated display, suchas a display monitor 600. The display monitor 600 of the control node102 can comprise a wire frame 605 of the tiled array display 500. Inother embodiments, the display monitor 600 of the control node 102 caninclude other visual cues for representing the borders of the individualdisplays 166 of each display node of the tiled array display. As shownin the lower portion of FIG. 6, the image 510 can be displayed both onthe tiled array display 500 and the display monitor 600 of the controlnode 102. The representation of the image 510 on the control nodedisplay 600 can be in the form of a reduced resolution version of theimage 510, a full resolution version, a thumbnail version, or otherversions.

The display monitor 600 of the control node can have any amount ofresolution. However, in some embodiments, the resolution of the display600 of the control node 102 will be significantly less than the totalresolution of the tiled array display 500 including all the individualdisplay units 100A, 100B, etc.

Thus, the wire frame 605 schematically representing the tiled arraydisplay 500 can be presented on the display 600 of the control node 102to provide an approximation of the proportions of the tiled arraydisplay 500. For example, if the tiled array display 500 provides atotal of 19,200 pixels wide by 10,800 pixels high, the display 600 ofthe control node 102 might include 1/10^(th) of that resolution, e.g.,1,920 pixels wide by 1,080 pixels high, or other resolutions.

The wire frame display can be the same or similar to the schematic wireframe representation illustrated in FIGS. 5A and 5B. The control node102 can be further configured to allow a user to manipulate theplacement of an image, such as the image 510, on the tiled arraydisplay. For example, a user can resize, reposition, rotate, and/oradjust color filtration or transparency of the one or more images. Insome embodiments, user interaction devices, such as a keyboard, mouse,3D mouse, speech recognition unit, gesture recognition device, Wiiremote, electronic pointing device, or haptic input device can be usedto manipulate the one or more images on the tiled array display 500. Anarrow pointer or other visual indicator can be displayed on the tiledarray display to allow for manipulation of the images via the userinteraction devices.

In some embodiments, the manipulation of the image 510 on the controlnode display 600 corresponds with the manipulation of the image on thetiled array display 500. For example, a user can provide inputs, forexample, by employing a mouse based control to drag and drop images onthe wire frame representation to change the position or orientation ofimages on the tiled array display 500. These inputs can be used by thecontrol node 102 to generate the messages (e.g., state messagescomprising image state values) described above.

The word “module,” as used herein, refers to logic embodied in hardwareor firmware, or to a collection of software instructions, possiblyhaving entry and exit points, written in a programming language, suchas, for example, Java, Lua, Objective-C, C or C++. A software module maybe compiled and linked into an executable program, installed in adynamic link library, or may be written in an interpreted programminglanguage such as, for example, BASIC, Perl, or Python. It will beappreciated that software modules may be callable from other modules orfrom themselves, and/or may be invoked in response to detected events orinterrupts. Software instructions may be embedded in firmware, such asan EPROM. It will be further appreciated that hardware modules may becomprised of connected logic units, such as gates and flip-flops, and/ormay be comprised of programmable units, such as programmable gate arraysor processors. The modules described herein are preferably implementedas software modules, but may be represented in hardware or firmware.Generally, the modules described herein refer to logical modules thatmay be combined with other modules or divided into sub-modules despitetheir physical organization or storage.

In addition the types of connections and couplings discussed above, anycoupling or connection discussed herein could be a local area network,wireless local area network, wide area network, metropolitan areanetwork, storage area network, system area network, server area network,small area network, campus area network, controller area network,cluster area network, personal area network, desk area network or anyother type of network.

Any of the computers, laptops, server, including the proxy server,control nodes, workstation, or other devices herein may be any type ofcomputer system. A computer system may include a bus or othercommunication mechanism for communicating information, and a processorcoupled with bus for processing information. Computer system may alsoincludes a main memory, such as a random access memory (RAM), flashmemory, or other dynamic storage device, coupled to bus for storinginformation and instructions to be executed by processor. Main memoryalso may be used for storing temporary variables or other intermediateinformation during execution of instructions to be executed byprocessor. Computer system may further include a read only memory (ROM)or other static storage device coupled to a bus for storing staticinformation and instructions for processor. A storage device, such as amagnetic disk, flash memory or optical disk, may be provided and coupledto bus for storing information and instructions.

The embodiments herein are related to the use of computer system for thetechniques and functions described herein in a network system. In someembodiments, such techniques and functions are provided by a computersystem in response to processor executing one or more sequences of oneor more instructions contained in main memory. Such instructions may beread into main memory from another computer-readable storage medium,such as storage device. Execution of the sequences of instructionscontained in main memory may cause a processor to perform the processsteps described herein. In alternative embodiments, hard-wired circuitrymay be used in place of or in combination with software instructions toimplement embodiments. Thus, embodiments are not limited to any specificcombination of hardware circuitry and software.

The term “computer-readable storage medium” as used herein, in additionto having its ordinary meaning, refers to any medium that participatesin providing instructions to a processor for execution. Such a mediummay take many forms, including but not limited to, non-volatile mediaand volatile media. Non-volatile media includes, for example, optical ormagnetic disks, such as a storage device. Volatile media includesdynamic memory, such as main memory. Transmission media includes coaxialcables, copper wire and fiber optics, including the wires that comprisebus.

Common forms of computer-readable media include, for example, a floppydisk, a flexible disk, hard disk, magnetic tape, or any other magneticmedium, a CD-ROM, DVD, any other optical medium, punch cards, papertape, any other physical medium with patterns of holes, a RAM, a PROM,and EPROM, a FLASH-EPROM, any other memory chip or cartridge.

Computer systems can send messages and receive data, including programcode, through the networks or other couplings. The received code may beexecuted by a processor as it is received, and/or stored in storagedevice, or other non-volatile storage for later execution. The term“broadcast” as used herein, in addition to having its ordinary meaning,may refer to any transmission of information over a network. The term“multicast” as used herein, in addition to having its ordinary meaning,can include broadcasting information to a subset of devices incommunication with a network.

Although the foregoing inventions have been described in terms of someembodiments, other embodiments will be apparent to those of ordinaryskill in the art from the disclosure herein. Moreover, the describedembodiments have been presented by way of example only, and are notintended to limit the scope of the inventions. Indeed, the novel methodsand systems described herein may be embodied in a variety of other formswithout departing from the spirit thereof. Accordingly, othercombinations, omissions, substitutions and modifications will beapparent to the skilled artisan in view of the disclosure herein. Thus,the present inventions are not intended to be limited by the preferredembodiments.

1. A tiled display system comprising: an array of display units incommunication with a network, each of the display units comprising animage display device and a processor, the image display devices beingarranged in a tiled layout; a control unit in communication with each ofthe display units via the network, the control unit configured totransmit to the display units, a message indicative of a state of adigital image; wherein the processor of each of the display units isconfigured to determine a respective portion of the digital image tooutput on its respective image display device based at least in part onthe message and to output the respective portions on the respectiveimage display devices so as to collectively present a unified display ofthe digital image on the array of display units.
 2. The system of claim1, wherein all of the display units are configured to store the digitalimage in the form of discrete tiles, a combination of the tiles formingthe entirety of the digital image.
 3. The system of claim 2, wherein allof the display units are configured to define the respective portion ofthe digital image as a subset of the discrete tiles, and to process anddisplay image data only from the subset of discrete tiles in response tothe message.
 4. The system of claim 1, wherein at least a plurality ofthe display units are configured to store the digital image in the formof discrete portions, a combination of the portions forming the entiretyof the digital image.
 5. The system of claim 4, wherein at least aplurality of the display units are configured to define the respectiveportion of the digital image as a subset of the discrete portions, andto process and display image data only from the subset of discreteportions in response to the message.
 6. The system of claim 1, whereinat least a plurality of the display units are configured to store aplurality of copies of the digital image, the plurality of copiescomprising data representing the digital image at different resolutions.7. The system of claim 6, wherein at least a plurality of the displayunits are configured to selectively process and display at least aportion of one of the plurality of copies based on the message. 8.(canceled)
 9. The system of claim 1, wherein the message comprises oneor more display parameters of the digital image.
 10. The system of claim9, wherein the one or more display parameters includes a location of areference pixel of the digital image on the array of display units. 11.The system of claim 9, wherein the one or more display parametersinclude a zoom level of the digital image.
 12. The system of claim 9,wherein the one or more display parameters include at least one of anindication of a size of the digital image, a pixel color value, and apixel transparency value.
 13. The system of claim 1, wherein the messagecomprises at least a portion of the digital image.
 14. (canceled) 15.(canceled)
 16. A method of presenting a large digital image on an arrayof display units, the array including a control unit communicating witheach of the display units over a network, each of the display unitscomprising a display monitor and a processor: transmitting a messagefrom the control unit to the display units, the message comprising atleast one display parameter of a large digital image; storing at least aportion of the large digital image on a local memory of each of thedisplay units; using a plurality of the display units in parallel toidentify the respective portions of the large digital image to displayon each of the respective display units, based at least in part on theat least one display parameter; using the respective ones of theplurality of the display units to process the identified respectiveportions of the large digital image; and using the respective ones ofthe plurality of the display units to output the respective portion ofthe large image on the respective display monitor of the display unitsin parallel, based at least in part on the at least one displayparameter.
 17. The method of claim 16, further comprising receiving thelarge digital image from an image data source in communication with thenetwork.
 18. The method of claim 16, wherein transmitting comprisestransmitting a location of a reference pixel of the large image on thearray.
 19. (canceled)
 20. (canceled)
 21. (canceled)
 22. The method ofclaim 16, further comprising storing sub-images of at least a portion ofthe large image on each of the display units.
 23. The method of claim22, wherein storing comprises storing multiple resolution layers of atleast a portion the large image.
 24. The method of claim 22, whereinstoring comprises storing a plurality of image blocks of at least aportion of the large image.
 25. A method of presenting a large digitalimage on an array of tiled display nodes, each of the tiled displayunits comprising a display monitor and a processor, the array alsoincluding a control unit communicating with each of the tiled displaynodes over a network, the method comprising: receiving a message from acontrol unit, the message comprising at least one display parameter of alarge digital image; determining a respective portion of the largedigital image to display on each of the display nodes in response to themessage received, based on the at least one display parameter; usingeach of the display nodes in parallel to process groups of one or morestored sub-images corresponding to the respective portions of the largedigital image; using each of the display nodes in parallel to output adisplay of the respective portions of the large digital image on therespective display monitors of each of the display nodes.
 26. The methodof claim 25, further comprising: receiving a second message from thecontrol unit, the second message comprising a second display parameter;and updating the display based on the second display parameter, thesecond display parameter having a different value than the at least onedisplay parameter.
 27. (canceled)
 28. (canceled)
 29. (canceled) 30.(canceled)
 31. (canceled)
 32. (canceled)
 33. (canceled)
 34. (canceled)35. (canceled)
 36. (canceled)
 37. (canceled)